On the Use and Misuse of Input-output Based Impact Analysis in Evaluation

Estimates of economic activity generated and jobs created that are derived using input-output analysis are often presented in program evaluations and confused with the benefit s resultin g from die program. Two such cases are presented as examples. We argue that for two main reason s this type of analysis con stitutes a misuse of input-output analysis. First, input-output estimates generated using the Keynesian closed versions of input-output models are biased upwards because they ignore the price and financial feedbacks that tend to reduce multipliers in macro-economic models. Second, and more important, it is inappropria te to consider induced effects resul ting from a particular program in isolation, because such effects can only be properly considered in the aggregate at the level of overall stabilization policy. In this paper we contend that cost-benefit analysis, with its assumption of full employment, is the most appropriate tool for analyzing the benefits resulting from particular programs. Inputoutput analysis should be confined to providing estimates of die industr ial or regional breakdown of the dir ect impact of a program or of the employment impacts of program spending. It should not be used to generate Keynesian multipliers. Resumé — Les estimations de 1'impact sur 1'activité éconononuque et 1'emplois faites avec 1'analyse intrantextrant se présentent souvent dans les évaluation des programmes et sont confuses avec les bénéfices du programme. Deux examples sont examinés içi. On prétends que ce genre d'analyse fasse un mauvais usage del'ana lyse intrant-extrant pour deux raisons principales. Premier, les estimations géncrée ut ilisant les modeles intrant-extrant de version kéynésienne et fermée sont predisposées a la hausse parce-qu'elles ign orent les rétroaction des prix et du secteur financier qui sont dans les modèles macroeconomiques. Deuxieme, et avec plus d'importance, il n'est pas propre a considérer les effect induits qui resultent d'un programme sauf dans Ie contexte de la polidque de stabiliza tion. On soutien que 1' analyse des couts ct bénéfices est le meilleur outil pour analyser les bénéfices d'un programme particulier. L'analyse intrant-extrant ne doit être utilisee que pour faire des estimation de 1'impact régional ou industiel et elle ne doit pas être utilisée pour le calcul des multiplicateur kéynésien. The Canadian Journal of Program Evaluation 2 ESTIMATES OF THE OUTPUT and employment impacts are often an important part of many program and project evaluations. The analytical framework frequently used to prepare these estimates is the family of inputoutput models developed and maintained by Statistics Canada . The output and employment mult ipliers derived from input-output models are used to tr anslate the direct impact of a program in to its total impact The total impact estimated, using the "closed" version of the input -output model , includes the output and employment generated by subsequent rounds of spending of the income created by the initial program expenditure. It thus reflects the traditional Keynesian multiplier taught in all in troductory economic textbooks. Frequently, the output and employment impacts of a program are treated as though they were "benefits" of the program or project They may be explici tly tabulated as benefits, or they may implicit ly be trea ted as such by bearing labels like "jobs created" or by being compared to the costs of the program. In this article, we argue that such estimated output and employment impacts in program or project evaluation are often used inappropriately. There are two reasons for this. First, many evaluators are not aware that the multipliers they use make very strong and often un tenable assumptions about macroeconomic impacts. The best evidence that is currently available suggests that multipliers derived from input-output models overestimate the impact of changes in government expenditures by ignoring the critical macroeconomic feedbacks that tend to reduce the multiplier over time. This dramatizes the need to separate the analyses of the microeconomic and the macroeconomic impacts of programs and projects. Second, many evaluators have an inadequate understanding of the principles of cost-benefit analysis. They thus tend to confuse the output and employment impacts of a program with i ts benefits. Th is tendency has been exacerbated by the emphasis in current program evaluation guidebooks on procedure, to the virtua l exclusion of discussion of the principles of cost-benefit analysis (e.g., Office of the Comptroller General, 1981 a and b). Section 2 of this article offers two examples of the misuse of output and employment impacts estimated using input-output techniques. Section 3 briefly describes the methodology of input-output models, discusses some of their limitations, and presents estimates of the multipliers derived from.them. Section 4 provides the details, from a macroeconomic point of view, of our criticism of the use of input-output multipliers. The main macroeconomic feedbacks that tend to dampen the response of the economy to government spending shocks are outlined, and estimates of multipliers from the main Canadian macroeconomic models are presented. Section 5 reviews the connection between economic impact analysis and cost-benefit analysis. It emphasizes a number of reasons that employment impacts cannot uncritically be considered benefits of a program or a project Section 6 gives our conclusions. EXAMPLES OF THE MISUSE OF IMPACT ANALYSIS As recent examples of the pervasive misuse of economic impact ana lysis, we consider a provincial position paper on housing policy in Ontario and a published evaluation of benefits from irrigation expenditures in Alberta. These two provincial examples were chosen because they have been published. Federal examples can be found in some of the unpublished program and project evaluations done in federal government departments. Most readers will thus recognize the phenomenon from their own experience. In a position paper issued in December 1985, the Ontario government introduced a number of initiatives to stimulate the construction of new housing (Ontario Ministry of Housing, 1985). These included interest-free loans to private rental developers, changes to the rent review system, increased social housing, and a strategy to stimulate the building industry. A table in the document "provides an overall picture of the estimated impact of the programs" (Ontario Ministry of Housing, 1985, pp. 29-30). fa aggregate, a provincial expenditure of $480 mil lion was expected to induce $5.2 billion of construction expenditures an d to "create" a lmost 200,000 job-years of employment Footnotes indicated that a multiplier of 2.2 person-years was used throughout the calculation. The The Canadian Journal of Program Evaluation 3 table leaves the clear impression that the programs can create employment at a cost to the government of $2,410 per job-year. As a second example. Kulshreshtha. Russell, Aycrs. and Palmer (1985) report on a study conducted for the Alberta Irrigation Projects Association . Here the goal was explicit ly to identify the major beneficiaries of irrigation activity. mput-output calculations showed that capital expenditures of about $348 million over the period 1985-89 would generate "benefits" of $415 million per year (Kulshreshthn et al.. 1985, pp. 7-8 ). Only 15% of these benefits would be received by water users. The remainder would be distr ibuted throughout the economies of Alber ta and the rest of Canada. Taken at face value, these results imply an annual return on investment of about 119%! What is wrong with these analyses? Our contention is that they, and many like them, confuse economic impacts with economic benefits. Even when this confusion is resolved, they exaggerate the impacts of programs and projects by comparing them to th e wrong benchmark and by using excessively high multipl iers to compute induced effects. In the next two sections we review how the multipliers are derived and how they compare to those estimated from large macroeconomic models. We then return to the relationship between benefits and impacts.